When Computer Vision Gazes at Cognition

Joint attention is a core, early-developing form of social interaction. It is based on our ability to discriminate the third party objects that other people are looking at. While it has been shown that people can accurately determine whether another person is looking directly at them versus away, little is known about human ability to discriminate a third person gaze directed towards objects that are further away, especially in unconstraint cases where the looker can move her head and eyes freely. In this paper we address this question by jointly exploring human psychophysics and a cognitively motivated computer vision model, which can detect the 3D direction of gaze from 2D face images. The synthesis of behavioral study and computer vision yields several interesting discoveries. (1) Human accuracy of discriminating targets 8{\deg}-10{\deg} of visual angle apart is around 40% in a free looking gaze task; (2) The ability to interpret gaze of different lookers vary dramatically; (3) This variance can be captured by the computational model; (4) Human outperforms the current model significantly. These results collectively show that the acuity of human joint attention is indeed highly impressive, given the computational challenge of the natural looking task. Moreover, the gap between human and model performance, as well as the variability of gaze interpretation across different lookers, require further understanding of the underlying mechanisms utilized by humans for this challenging task.Comment: Tao Gao and Daniel Harari contributed equally to this wor

Do You See What I Mean? Visual Resolution of Linguistic Ambiguities

Understanding language goes hand in hand with the ability to integrate complex contextual information obtained via perception. In this work, we present a novel task for grounded language understanding: disambiguating a sentence given a visual scene which depicts one of the possible interpretations of that sentence. To this end, we introduce a new multimodal corpus containing ambiguous sentences, representing a wide range of syntactic, semantic and discourse ambiguities, coupled with videos that visualize the different interpretations for each sentence. We address this task by extending a vision model which determines if a sentence is depicted by a video. We demonstrate how such a model can be adjusted to recognize different interpretations of the same underlying sentence, allowing to disambiguate sentences in a unified fashion across the different ambiguity types.Comment: EMNLP 201

Education Bill: Committee Stage Report. Research paper 11/37

"This is an account of the House of Commons Committee Stage of the Education Bill. It complements Research Paper 11/14, prepared for the Commons Second Reading debate... The Bill, as amended in Public Bill Committee, was published as Bill 180.

Moth-inspired navigation algorithm in a turbulent odor plume from a pulsating source

Some female moths attract male moths by emitting series of pulses of pheromone filaments propagating downwind. The turbulent nature of the wind creates a complex flow environment, and causes the filaments to propagate in the form of patches with varying concentration distributions. Inspired by moth navigation capabilities, we propose a navigation strategy that enables a flier to locate a pulsating odor source in a windy environment using a single threshold-based detection sensor. The strategy is constructed based on the physical properties of the turbulent flow carrying discrete puffs of odor and does not involve learning, memory, complex decision making or statistical methods. We suggest that in turbulent plumes from a pulsating point source, an instantaneously measurable quantity referred as a "puff crossing time", improves the success rate as compared to the navigation strategy based on "internal counter" that does not use this information. Using computer simulations of fliers navigating in turbulent plumes of the pulsating point source for varying flow parameters: turbulent intensities, plume meandering and wind gusts, we obtained trajectories qualitatively resembling male moths flights towards the pheromone sources. We quantified the probability of a successful navigation as well as the flight parameters such as the time spent searching and the total flight time, with respect to different turbulent intensities, meandering or gusts. The concepts learned using this model may help to design odor-based navigation of miniature airborne autonomous vehicles